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i 

ADVANCED CONTROL DEVICE FOR HOME ENTERTAINMENT UTILIZING 

THREE DIMENSIONAL MOTION TECHNOLOGY 

5 The present invention relates to the control of 

home entertainment devices and applications, and more 
particularly, to a method and system for controlling and 
transferring data to home entertainment devices by 
manipulating a control device. 

10 Hand-held devices, such as remote controls devices, 

are typically used to control consumer electronic devices, 
such as televisions and gaming machines. As the hand-held 
devices and consumer electronic devices have become more 
sophisticated, new techniques for inputting commands to the 

15 hand-held devices have been developed. These techniques 
include methods that detect the orientation of a hand-held 
device to generate a command. For example, U.S. Patent 
Numbers 4,745,402 and 4,796,019 disclose methods for 
controlling the position of a cursor on a television. U.S. 

20 Patent Number 6,603,420 discloses a remote control device 
that detects the direction of movement of the remote control 
device to control, e.g., the channel and volume selection of 
a television. 

The ability of these hand-held devices to hold data 
25 and the development of more sophisticated capabilities in the 
consumer electronic devices has created new challenges for 
controlling these consumer electronic devices. For example, 
it is often necessary to transfer data from the hand-held 
device to the consumer electronic device or vice versa. The 
30 hand-held device should also provide a natural, efficient 
mechanism for indicating that an action, such as a data 
transfer, is to be performed. A need therefore exists for an 
improved hand-held device that is capable of efficiently 
generating commands and transferring data to or from consumer 
35 electronic devices. 
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An apparatus and method . are disclosed for 
generating commands and transferring data between a hand-held 
device and a base device (including consumer electronic 
equipment). The hand-held device is capable of detecting . the 
S motion of the hand-held device itself, interpreting the 
motion as a command, and executing or transferring the 
command. The motion of the device can include gestures made 
by the usfer while holding the device, such as the motion of 
throwing the hand-held device toward a base device, as a user 

10 would do when swinging a tennis racket. The commands 
generated by the user range from basic on/off commands to 
complex processes, such as the, transfer of data. 

In one embodiment, the user can train the device to 
learn new motions associated with existing or new commands. 

15 For example, the user can make the motion of throwing the 
hand-held device toward the base device. The hand-held 
device analyzes the basic components of the motion to create 
a motion model such that the motion can be uniquely 
identified in the future. 

20 A more complete understanding of the present 

invention, as well as further features and advantages of the 
present invention, will be obtained by reference to the 
following detailed description and drawings. 

FIG. 1 shows an exemplary hand-held device of the 

25 present invention; 

FIGS. 2A-B illustrate gestures that are interpreted 
as commands by the hand-held device of FIG. 1; 

PIG. 3 is a schematic block diagram of the hand- 
held device of FIG. 1; 

30 'FIG. 4 illustrates an exemplary embodiment of a 

motion detection subsystem; 

FIG. 5 is a flowchart describing an exemplary 
implementation of the system process of the hand-held device 
of FIG. 1; 

2 
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FIG. 6 is a flowchart describing an exemplary 
implementation of a motion training process; FIG. 7 is a 
flowchart describing an exemplary implementation of a motion 
detection process; and 
5 FIG. 8 is a graph illustrating the motion model of 

a throwing motion based on the expected acceleration in each 
of three perpendicular planes . 

FIG. 1 shows an exemplary hand-held device 300 of 
the present invention, discussed further below in conjunction 

10 with FIG. 3, such as the Philips Super Pronto, modified in 
• accordance with the features of the present invention. The 
hand-held device 300 is capable of detecting motion of the 
hand-held device 300, interpreting the detected motion as one 
or more commands, and executing or transferring the 

15 command (s) . 

FIGS. 2A-B illustrate gestures that a user can make 
using the hand-held device 300, FIG. 2A, for example, shows 
a user 201 making the gesture of throwing the device 300 
toward a base device, such as television 210. FIG. 2B shows 

20 a user making the gesture of pouring from the device 300 into 
a base device, such as television 210. The gesture and 
associated motion indicate that the user 201 would like to 
transfer data from- the hand-held device 300 to the television 
210. In this case, the user would first locate and identify 

25 the data (e.g. a picture or music) and then make the gesture 
toward the base device, The data could be identified, for 
instance, by selecting an item from of a list displayed on 
the hand-held device 300. The data would then be 

transferred. In addition, if the data is a picture, it could 

30 be (optionally) displayed on the television or, if the data 
is music, it could be (optionally) played through the 
speakers. Other gestures include making a pulling motion 
(not shown) directed from a base device towards the user. In 
this case, the gesture would indicate that the identified 

35 data should be transferred to the hand-held device 300. The 

3 
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data would then be retrieved from either the base device 
itself, or from another - device (e.g. a server). Since there 
are a number of base devices 210 through 214 located in the 
area of the user 201, the hand-held device 3 00 has the 
ability to identify which device 210-214 should receive the 
data being transferred (as described in more detail 
below). PIG. 3 is a schematic block diagram of an exemplary 
hand-held device 300 of the present invention. As is known 
in the art, the methods and apparatus discussed herein may be 
distributed as an article of manufacture that itself 
comprises a computer- readable medium having computer- readable 
code means embodied thereon. The computer- readable program 
code means is operable, in conjunction with a computer system 
such as central processing unit 301, to carry out all or some 
of the steps to perform the methods or create the apparatuses 
discussed herein. The computer -readable medium may be a 
recordable medium (e.g., floppy disks, hard drives, compact 
disks, or memory cards) or may be a transmission medium 
(e.g., a network comprising fiber-optics, the world-wide web, 
cables, or a wireless channel using time-division multiple 
access, code-division multiple access, or other radio- 
frequency channel) . Any medium known or developed that can 
store information suitable for use with a computer system may 
be used. The computer- readable code means is any mechanism 
for allowing a computer to read instructions and data, such 
as magnetic variations on a magnetic medium or height 
variations on the surface of a compact disk. 

Memory 302 will configure the processor 301 to 
implement the methods, steps, and functions disclosed herein. 
The memory 3 02 could be distributed, or local and the 
processor 301 could be distributed or singular. The memory 
302 could be implemented as an electrical, magnetic or 
optical memory, or any combination of these or other types of 
storage devices. The term "memory" should be construed 
broadly enough to encompass any information able to be read 
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from or written to an address in the addressable space 
accessed by processor 301. 

As shown in FIG. 3, the memory 302 includes motion 
model database 3 03, system process 500, discussed further 

* 

5 below in conjunction with FIG. 5, motion training process 
600, discussed further below in conjunction with FIG. 6, and 
motion detection process 700 , discussed further below in 
conjunction with FIG. 7. Hand-held device 300 also includes 
motion detection subsystem 400, discussed further below in 
10 conjunction with FIG. 4, radio frequency (RF) communication 
subsystem 305, and infrared detection subsystem (IDS) 310. 

The RF communication subsystem 3 05 provides 
communication between the handheld device 300 and one or more 

« 

base devices 210-214 in a known manner. For example, the RF 

15 communication subsystem 305 may utilize the IEEE 802.11 
standard for wireless communications or any extensions 
thereof. The IDS 310 emits infrared light in a directional 
manner in order to signal a base device 210-214 that it 
should execute the command being transmitted by the device 

20 300. Only the base device 210-214 that detects the infrared 
signal should execute the transmitted command. The command 
is transferred to the base device 210-214 via the RF 
communication subsystem 305 in a known manner. In an 
alternative embodiment, the command may be transferred by 

25 modulating the infrared signal {utilizing, , for example, the 
IR Blaster standard) in a known manner. 

FIG. 4 illustrates an exemplary embodiment of 
motion detection subsystem 400. Motion detection subsystem 
400 contains x-axis accelerometer sensor 410, y-axis 

30 accelerometer sensor 411, z-axis accelerometer sensor 412, 

and corresponding analog to digital converters • 415 , 416, 417. 
Accelerometer sensors 410, 411, 412 detect the acceleration 
of the device 300 along the x-axis, y-axis, and z-axis, 
respectively. The accelerometer sensors 410, 411, 412 may be 

35 embodied, for example, using the 3D Motion Sensors 
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commercially available from NECToxin of Union City, 
California. Analog to digital converters 415 , 416, 417 
convert the acceleration (s) detected by accelerometer sensors 
410, 411, 412, respectively, to a digital form that cam be 
5 read by processor 301. In alternative embodiments, other 
components, including stress-sensitive resistive elements, 
tilt sensors, and magnetic direction sensors, may be utilized 
to determine the position, orientation and/or speed of 
movement of the device 300. 

10 FIG. 5 illustrates an exemplary embodiment of 

system process 500. System process 500 initially waits for a 
command to be entered during step 505. ' If, during step 505, 
a user enters a training command, the system process 500 
executes step 510 where motion training process \ 600 is 

15 called. If, during step 505, a user makes a gesture or 
motion indicative of a command, the system process 500 
executes step 515 where motion detection process 700 is 
called. Upon completion of the called processes 600, 700, 
system process 500 returns to step 505 to wait for the entry 

20 of a new command. 

FIG. 6 illustrates an exemplary embodiment of 
motion training process 600. Motion training process 600 
learns new gestures and motions demonstrated by a user to be 
used for identifying existing or new commands. For instance, 

25 a user 201 may want to train the device 300 to detect a 
throwing, motion, such as the motion of throwing the device 
3 00 toward a television 210. The user first presses a switch 
on the hand-held device 300 to indicate that a new gesture is 
to be created. (Alternatively, the user can train the hand- 

30 held device 300 to interpret a motion as an indication that 
the training process should be executed.) Motion training 
process 600 initially waits for motion to lye detected by one 
or more of the accelerometer sensors 410, 411, 412 (step 601) 
and then records the motion detected by the sensors 410, 411, 

35 412 by periodically sampling and storing data read from 
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analog to digital converters 415, 416, 417 (step 605). After 
each set of samples have been read during sampling step 605, 
a test is made to determine if no motion has been detected 
for a specified period of time indicating that the gesture or 
5 motion has been completed (step 608) . If motion is detected 
during step 608, then step 605 is repeated to read the next 
set of samples; otherwise, motion training process 600 i 
creates and stores a model of the motion captured during step 
610. The motion model is created in a known manner For 

10 example, the following publications describe methods for 
analyzing, comparing and modeling motions and gestures: Ho- 
Sub Yoon, Jung Soh, Younglae J. Bae and Hyun Seung Yang, Hand 
Gesture Recognition Using Combined Features of Location, 
Angle and Velocity, Pattern Recognition, Volume 34, Issue 7, 

15 2001, Pages 1491-1501; Cristopher Lee and Yangsheng Xu, 
Online, Interactive Learning of Gestures for Human/Robot 
Interfaces, The Robotics Institute, Carnegie Mellon 
University, Pittsburgh, IEEE International Conference on 
Robotics and Automation, Minneapolis, 1996; Mu-Chun Su, Yi- 

20 Yuan Chen, Kuo-Hua Wang, Chee-Yuen Tew and Hai Huang, 3D Arm 
Movement Recognition Using Syntactic Pattern ' Recognition, 
Artificial Intelligence in Engineering, Volume 14 Issue 2, 
April 2000, Pages 113-118; Ari Y, Benbasat and Joseph A. 
Paradiso, An Inert ial Measurement Framework for Gesture 

25 Recognition and Applications ) MIT Media Laboratory, 
Cambridge, 2001; and Mu-Chun Su, Yi-Yuan Chen, Kuo-Hua Wang, 
Chee-Yuen Tew and Hai Huang, 3D Arm Movement Recognition 
Using Syntactic Pattern Recognition, Artificial Intelligence 
in Engineering, Volume 14, Issue 2, April 2000, Pages 113- 

3 0 118, each incorporated by reference herein. 

The created model will be used to interpret 
future gestures and motions made by the user 201. During 
step 615, the model created during step 610 is assigned a 
command or process that is to be executed when the motion 

35 associated with the model is detected. The command to be 

7 
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executed is identified utilizing well known methods , for 
instance, pressing a switch on the hand-held device 300 
associated with the command or entering a code associated 
with the command on a keypad. In an alternative embodiment, 
5 the user could enter (record) a series of commands by 
performing the actions on the system (e.g., on the touch 
screen) , similar to recording a macro in MS Word. The series 
of commands can then be associated to a single gesture. The 
assigned command or process is stored with the associated 

10 motion model in the motion model database 303. 

FIG. 7 illustrates an exemplary embodiment of 
motion detection process 700. Motion detection process 700 
interprets gestures and motions made by a user 201 to 
determine the command(s) that are to be executed. For 

15 instance, if the user 201 makes the motion of throwing the 
hand-held device 3 00 towards the television 210, the hand- 
held device 300 will interpret the gesture as a command to 
transfer data from the device 300 to the television 210. 
Motion detection process 700 initially records the motion 

20 detected by the accelerometer sensors 410, 411, 412 by 
periodically sampling and storing the data read from analog 
to digital converters 415, 416, 417 (step 705) . After each 
set of samples have been read during sampling step 705, a 
test is made to determine if no motion has been detected for 

25 a specified period of time indicating that the gesture or 
motion has been completed (step 708) . If motion is detected 
during step 70 8, then step 705 is repeated to read the next 
set of samples; otherwise, motion detection process 700 
compares the data collected during step 705 to the motion 

30 models stored in the device 300 (step 710) . During step 710, 
a score is generated for each model comparison. The command 
or process associated with the model that attained the 
highest score during step 710 is then executed during step 
715. For example, if the model with the highest score was 

35 the "throwing motion" model, then a data transfer process 

8 
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(not shown) would be executed in a known manner. The data 
transfer process can be accomplished/ for example, utilizing 
the 802.11 standard in a well known manner. During step 72 0/ 
the IDS 310 is also activated, thereby causing an infrared 
5 signal to be emitted in the direction of the throwing motion. 
Only the base device 210-214 that detects the infrared signal 
will receive the data transferred via the RF communication 
subsystem 305. 

FIG. 8 shows an exemplary motion model representing 

10 the throwing motion of FIG. 2A. As is illustrated, the z- 
axis accelerometer indicates that the motion is in the x-y 
plane (no motion along the z-axis) . As indicated by the x- 
axis accelerometer, the motion shows a quick acceleration 
along the x-axis, a peak speed at the halfway point of the 

15 motion, and an increasing deceleration as the motion is 
completed. A similar, but smaller, action is occurring along 
the y-axis. 

It is to be understood that the embodiments and 
variations shown and described herein are merely illustrative 
20 of the principles of this invention and that various 
modifications may be implemented by those skilled in the art 
without departing from the scope and spirit of the invention. 

* 

25 
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CLAIMS 

1. An apparatus for controlling a base device, 
comprising: 

5 a memory; and 

at least one processor, coupled to the memory, 
operative to: 

detect a motion of said apparatus ; 

interpret said motion to identify a command that 
10 triggers a transfer of data between said apparatus and said 
base device; and 

execute said command. 

2. The apparatus of claim 1, wherein said execute said 
15 command operation includes transferring a second command to 

said base device. 

3. The apparatus of claim 1, wherein said detected 
motion is a throwing motion. 

20 

4. The apparatus of claim 1, wherein said detected 
motion is a pouring motion. 

5. The apparatus of claim 1, wherein said detected 
25 motion is a pulling motion directed from said base device. 

6. The apparatus of claim 1, further operative to add 
one or more new commands by detecting and recording a 
demonstration motion, 

30 

7. The apparatus of claim 6, further operative to 
create a motion model from said recorded demonstration 
motion. 

10 
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8. The apparatus of claim 7, further operative to 
assign said one or more new commands to said motion model. 

9. The apparatus of claim 1, further comprising three 
dimensional motion sensors for performing said motion 
detection operation. 

10. The apparatus of claim 1, further comprising one or 
more motion models, wherein each of said one or more motion 
models is assigned a command. 

11. The apparatus of claim 10, wherein said interpret 
said motion operation is performed by comparing said detected 
motion to one or more of said one or more motion models. 

12. A method for controlling a base device, comprising: 

detecting a motion of said apparatus; 
interpreting said motion to identify a command that 

triggers a transfer of data between said apparatus and said ' 
base device; and 

executing said command. 

13. The method of claim 12, wherein said executing said 
command step includes transferring a second command to said 

25 base device. 

14. The method of claim 12, wherein said detecting 
motion step is a throwing motion. 

30 15. The method of claim 12, wherein said detecting 

motion step is a pouring motion. 

16. The method of claim 12, wherein said detecting 

motion step is a pulling motion directed from said base 
35 device. 

11 
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1 7 - The method of claim 12, further comprising the step 
of adding, one or more new commands hy detecting and recording 
a demonstration motion. 

5 

18- The method of claim 17-, further comprising the step 
of creating a motion model from said recorded demonstration 
motion. 

10 19. The method of claim 18, further comprising the step 

of assigning said one or more new commands to said motion 
model . 

20. The method of claim 12, wherein said interpreting 
15 said motion step is performed by comparing said detected 

motion to one or more motion models. 

21. An article of manufacture for controlling a base 
device, comprising: 

20 a machine readable medium containing one or more 

programs which when executed implement the steps of: 

detecting a motion of said apparatus; 
interpreting said motion to identify a command that 
triggers a transfer of data between said apparatus and said 
25 base device; and 

executing said command. 



30 
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ABSTRACT 

A hand held device for generating commands and 
transferring data between the hand-held device and a base 
device (including consumer electronic equipment) . The hand- 
held device detects the motion of the device itself, 
interpreting the motion as a command, and executing or 
transferring the command. The motion of the device can 
include gestures made by the user while holding the device, 
such as the motion of throwing the hand- held device toward a 
base device. The commands generated by the user range from 
basic on/off commands to complex processes/ such as the 
transfer of data. In one embodiment, the user can train the 
device to learn new. motions associated with existing or new 
commands. The hand-held device analyzes the basic components 
of the motion to create a motion model such that the motion 
can be uniquely identified in the future. 
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